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DETAILED ACTION 
Response to Amendment 

1 . This Office Action is responsive to the amendment received 6 October 2009. 

2. Claims 10, 15 and 16 are currently amended, claims 1 1-14 are as previously presented 

and claims 1-9 were previously cancelled. 

3. The AppUcants' amendment has necessitated the new grounds of rejection that follow. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained iliough the eniion is noi ideiuieally diselosed or described as set forth in 
seetion 102 of this title, if the dit't'erenees between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 
(1966), that are applied for establishing a background for determining obviousness under 35 
U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness 
or nonobviousness. 

4. Claims 1-9 (Cancelled). 

5. Claims 10 and 13-16 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Williamson et al. (U. S. Patent Application Publication 2004/0104935 Al, already of record, 
hereafter '935) and further in view of Latham (U. S. Patent 5,803,738, hereafter '738). 
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6. In regard to claim 10 (Currently Amended), Williamson teaches an image processing 
apparatus for composting an image of a virtual object and an image of a physical space to 
generate a mixed reality image and causing a HMD to display the mixed reality image ('935; 
abstract; fig. 6; ^ 0224), comprising: a database which holds data used for generating the image 
of the virtual object ('935; fig. 1; 3D model extraction and hard disk storage; 0060; ^ 0069- 
0074); an image capturing unit which is attached to the HMD and captures the image of the 
physical space ('935; HMD with a camera attached, T| 0224); a first measurement unit which 
measures a position and orientation of the HMD ("935; camera and targets; abstract; T| 0021- 
0024); but does not teach an object manipulation unit which is used by a user wearing the HMD 
in order to operate a position and orientation of the virtual object; a second measurement unit 
which measures a position and orientation of said object manipulation unit; an operation panel 
which can be operated by the user, arranged at a position in the physical space within a viewing 
field of said image capturing unit attached to the HMD , displays an operation panel image used 
for editing the virtual object, and is capable of receiving a user instruction of editing the virtual 
object input bv the user ; an operation panel image generation unit which generates the operation 
panel image by using the data held in said database, and outputs the generated operation panel 
image to said operation panel; and continues teaching a rendering unit ('935; ^ 0070-0074) 
which updates the data held in said database according to the user instruction received via said 
operation panel and the measurement result of said second measurement unit ('935; fig. 1; 3D 
model extraction and hard disk storage; H 0060; H 0069-0074), and renders ('935; 1 0100-01 14), 
by using the updated data, the image of the virtual object according to the measurement results of 
said first and second measurement units; and a composition unit which composites the image of 
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the virtual object rendered by said rendering unit and the image of the physical space captured by 
said image capturing unit to generate the mixed reality image ('935; fig. 17, element 1707; ^ 
0297-0299); and a HMD which displays the mixed reality image generated by said composition 
unit ('935; fig. 17, element 1707; H 0297-0299). Latham, working in the same field of endeavor, 
however, teaches an object manipulation unit which is used by a user wearing the HMD in order 
to operate a position and orientation of the virtual object ('738; fig. 1, element 16, data glove; 
col. 3, In. 10-15); a second measurement unit which measures a position and orientation of said 
object manipulation unit ('738; fig. 1, element 18, tracker; col. 3, In. 10-15); an operation panel 
which can be operated by the user, arranged at a position in the physical space within a viewing 
field of said image capturing unit attached to the HMD ('738; fig. 1, element 12, panel; col. 3, In. 
21-37), displays an operation panel image used for editing the virtual object, and is capable of 
receiving a user instruction of editing the virtual object input by the user ('738; fig. 5; col 4, In. 
53 through col. 5, In. 42); an operation panel image generation unit which generates the 
operation panel image by using the data held in said database ('738; fig. 4, element 60, 
computer), and outputs the generated operation panel image to said operation panel ('738; fig. 4, 
element 70, output to image generator interface) for the benefit of allowing the user of the 
simulation or virtual reality environment to directly control the virtual object(s) within the mixed 
reality environment. It would have been obvious to one of ordinary skill in the art at the time of 
the invention to have combined the virtual object controlling and control panel teachings of 
Latham with the VR teachings of Williamson using a head mounted display coupled with a head 
mounted camera and head/body location techniques for the benefit of allowing the user of the 
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immersion simulation or virtual reality environment to directly control the virtual object(s) 
presented within the mixed reality environment. 

7. Regarding claim 13 (Previously Presented), Williamson and Latham teach the apparatus 
according to claim 10 and Latham further teaches wherein said operation panel includes a 

display device and an operation device ('738; fig. 1, element 12, panel; col. 3, In. 21-37), 
wherein the display device displays the operation panel image, and wherein the operation device 
is used for inputting the user instruction ('738; fig. 1, element 12, panel; col. 3, In. 21-37). 

8. In regard to claim 14 (Previously Presented), Williamson and Latham teach the apparatus 
according to claim 10 and Williamson further teaches wherein the HMD can present the mixed 
reality image to two eyes of a user who wears the HMD ('935; fig. 16, element 1603; ^ 0293- 
0295). 

9. Regarding claim 15 (Currently Amended) Williamson teaches an image processing 
method of composting an image of a virtual object and an image of a physical space to generate a 
mixed reality image and causing an HMD to display the mixed reality image ('935; abstract; fig. 
6; H 0224), comprising the steps of: holding data used for generating the image of the virtual 
object in a database ('935; fig. 1; 3D model extraction and hard disk storage; T| 0060; T| 0069- 
0074); providing an image capturing unit, which is attached to the HMD ('935; HMD with a 
camera attached, ^ 0224), to capture the image of the physical space ('935; abstract; fig. 6; ^ 
0224); measuring a position and orientation of the HMD with a first measurement unit ("935; 
camera and targets; abstract; Tf 0021-0024); but does not teach operating an object manipulation 
unit, by a user wearing the HMD , in order to position and orient the virtual object; measuring a 
position and orientation of the object manipulation unit with a second measurement unit; 
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receiving a user instmction of editing the virtual object, input by the user using an operation 
panel, wherein the operation panel is arranged at a position in the physical space within a 
viewing field of the image capturing unit attached to the HMD and displays an operation image 
used for editing the virtual object: generating the operation panel image by using the data held in 
the database with an operation panel image generation unit and outputting the generated 
operation panel image to the operation panel; and continues teaching updating the data held in 
the database according to the user instruction received via the operation panel and the 
measurement result of the second measurement unit ('935; fig. 1; 3D model extraction and hard 
disk storage; 1 0060; 1 0069-0074), and rendering ('935; ^ 0100-01 14), by using the updated 
data, the image of the virtual object according to the measurement results of the first and second 
measurement units; compositing the rendered image of the virtual object and the captured image 
of the physical space to generate the mixed reality image ('935; fig. 17, element 1707; T| 0297- 
0299); and displaying the generated mixed reality image on the HMD ('935; fig. 17, element 
1707; TI 0297-0299). Latham, working in the same field of endeavor, however, teaches an object 
manipulation unit which is used by a user wearing the HMD, in order to position and orient the 
virtual object ('738; fig. 1, element 16, data glove; col. 3, In. 10-15); measuring a position and 
orientation of the object manipulation unit with a second measurement unit ('738; fig. 1, element 
18, tracker; col. 3, In. 10-15); receiving a user instruction of editing the virtual object, input by 
the user using an operation panel, wherein the operation panel is arranged at a position in the 
physical space within a viewing field of the image capturing unit attached to the HMD ('738; fig. 
1, element 12, panel; col. 3, In. 21-37) and displays an operation image used for editing the 
virtual object ('738; fig. 5; col 4, In. 53 through col. 5, In. 42)^ generating the operation panel 
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image by using the data held in the database ('738; fig. 4, element 60, computer) with an 
operation panel image generation unit and outputting the generated operation panel image to the 
operation panel ('738; fig. 4, element 70, output to image generator interface) for the benefit of 
allowing the user of the simulation or virtual reality environment to directly control the virtual 
object(s) within the mixed reality environment. It would have been obvious to one of ordinary 
skill in the art at the time of the invention to have combined the virtual object controlling and 
control panel teachings of Latham with the VR teachings of Williamson using a head mounted 
display coupled with a head mounted camera and head/body location techniques for the benefit 
of allowing the user of the immersion simulation or virtual reality environment to directly control 
the virtual object(s) presented within the mixed reality environment. 
10. In regard to claim 16 (Currently Amended), Williamson teaches computer-readable 
storage medium encoded with a computer program ('935; fig. 16; computer and memory) for an 
image processing method of composting an image of a virtual object and an image of a physical 
space to generate a mixed reality image and causing an HMD to display the mixed reality image 
('935; abstract; fig. 6; ^ 0224), comprising the steps of: holding data used for generating the 
image of the virtual object in a database ('935; fig. 1; 3D model extraction and hard disk storage; 
TI 0060; TI 0069-0074); providing an image capturing unit, which is attached to the HMD ('935; 
HMD with a camera attached, ^ 0224), to capture the image of the physical space; measuring a 
position and orientation of the HMD with a first measurement unit ("935; camera and targets; 
abstract; ^ 0021-0024); but does not teach operating an object manipulation unit, by a user 
wearing the HMD , in order to position and orient the virtual object; measuring a position and 
orientation of the object manipulation irnit with a second measurement irnit; receiving a user 
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instruction of editing the virtual object, input by the user using an operation panel, wherein the 
operation panel is arranged at a position in the physical space within a viewing Field of the image 
capturing unit attached to the HMD and displays an operation image used for editing the virtual 
object; generating the operation panel image by using the data held in the database with an 
operation panel image generation unit and outputting the generated operation panel image to the 
operation panel; and continues teaching updating the data held in the database according to the 
user instruction received via the operation panel and the measurement result of the second 
measurement unit, and rendering, by using the updated data, the image of the virtual object 
according to the measurement resuhs of the first and second measurement units; compositing the 
rendered image of the virtual object and the captured image of the physical space to generate the 
mixed reality image ('935; fig. 17, element 1707; T| 0297-0299); and displaying the generated 
mixed reality image on the HMD ('935; fig. 17, element 1707; H 0297-0299). Latham, working 
in the same field of endeavor, however, teaches operating an object manipulation unit, by a user 
wearing the HMD ('738; fig. 1, element 16, data glove; col. 3, In. 10-15), in order to position and 
orient the virtual object; measuring a position and orientation of the object manipulation unit 
with a second measurement unit ('738; fig. 1, element 18, tracker; col. 3, In. 10-15); receiving a 
user instruction of editing the virtual object, input by the user using an operation panel ('738; fig. 
1, element 12, panel; col. 3, In. 21-37) , wherein the operation panel is arranged at a position in 
the physical space within a viewing field of the image capturing unit attached to the HMD and 
displays an operation image used for editing the virtual object ('738; fig. 1, element 12, panel; 
col. 3, In. 21-37); generating the operation panel image by using the data held in the database 
with an operation panel image generation unit ('738; fig. 4, element 60, computer) and 
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outputting the generated operation panel image to the operation panel ('738; fig. 4, element 70, 
output to image generator interface) for the benefit of allowing the user of the simulation or 
virtual reality environment to directly control the virtual object(s) within the mixed reality 
environment. It would have been obvious to one of ordinary skill in the art at the time of the 
invention to have combined the virtual object controlling and control panel teachings of Latham 
with the VR teachings of Wilhamson using a head mounted display coupled with a head 
mounted camera and head/body location techniques for the benefit of allowing the user of the 
immersion simulation or virtual reality environment to directly control the virtual object(s) 
presented within the mixed reality environment. 

1 1 . Claims 11-12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Williamson 
et al. (U. S. Patent Application Publication 2004/0104935 Al, already of record, hereafter '935) 
as applied to claims 10 and 13-16 above and in view of Latham (U. S. Patent 5,803,738, 
hereafter '738) as applied to claims 10 and 13-16 above, and further in view of Lescinsky et al. 
("Interactive Scene Manipulation in the Virtue3D System," ACM Weg3D'02, pp. 127-135, 
February 24-28, 2002, Tempe, Arizona, ACM 1-581 13-468-1/02/0002, already of record, 
hereafter '468). 

12. Regarding claim 1 1 (Previously Presented), Williamson and Latham teach the apparatus 
according to claim 10 but do not teach wherein the image of the virtual object is generated on the 
basis of 3D CAD data of the virtual object, and said operation panel displays an assembly tree 
based on the 3D CAD data. Lescinsky, working in the same field of endeavor, however, teaches 
wherein the image of the virtual object is generated on the basis of 3D CAD data of the virtual 
object, and said operation panel displays an assembly tree based on the 3D CAD data ('468; fig. 



Application/Control Number: 10/594,114 Page 10 

Art Unit: 2628 

4, pg. 133, § 8-9) for the benefit of providing tools to allow easy to use scene authoring and 
manipulation of content within the virtual 3D mixed reality environment. It would have been 
obvious to one of ordinary skill in the art at the time of the invention to have combined the VR 
apparatus teachings of Williamson and Latham with the explicit VRML tree teachings of 
Lescinsky for benefit of providing tools to allow easy to use scene authoring and manipulation of 
content within the virtual 3D mixed reality environment. 

13. In regard to claim 12 (Previously Presented), Lescinsky fiirther teaches wherein a part, 
which is obtained by enlarging a designated part of the assembly tree included in the operation 
panel image, includes a component name contained in the assembly tree ('468; fig. 4, pg. 133, § 
8-9). 

Response to Arguments 

14. Applicants' arguments with respect to claims 10-16 have been considered but are moot in 
view of the new ground(s) of rejection. 

Conclusion 

The following prior art, made of record, was not relied upon but is considered pertinent to 

applicant's disclosure: 

US 6396497 Bl Computer user interface with head motion input - Teaches HMD in 

a virtual environment with a user operable control panel within the 
users field of view. 



THIS ACTION IS MADE FINAL. Apphcant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Edward Martello whose telephone number is (571) 270-1883. 
The examiner can normally be reached on M-F 7:30-5:00 EST. 

If attempts to reach the examiner by telephone are unsuccessfiil, the examiner's 
supervisor, Xiao Wu can be reached on (571) 272-7761 . The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an apphcation may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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